Skip to content

fix: report a close or cancel during a wait as OperationStateError - #609

Closed
vuanhphung wants to merge 6 commits into
mainfrom
fix/close-during-wait-reports-closed
Closed

vuanhphung wants to merge 6 commits into
mainfrom
fix/close-during-wait-reports-closed

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Closing a session while one of its queries is running should make the code waiting on that query (fetchAll(), fetchChunk(), finished()) fail with a closed error. On Thrift it often fails with the transport error instead:

THTTPException: Received a response with a bad HTTP status code: 409: ABORTED: A concurrent GetOperationStatus request updated the command. Please retry.

session.close() sends CloseOperation while the wait's GetOperationStatus is still in flight. The server answers that poll with HTTP 409, or sometimes with HTTP 404 RESOURCE_DOES_NOT_EXIST for the released statement, and DBSQLOperation passes the error through, even though the client itself closed the operation. The "Please retry" text points callers the wrong way. The databricks-driver-test CONCURRENT-004 test (databricks/databricks-driver-test#2082) reproduces it live against main (8ccddd0): the proxy recorded the polls as 200,409 and the waiter got the 409. In #2082's runs the failed poll was a 409 in 10 of 13 and a 404 in the other 3. A cancel at the same point already gets CANCELED_STATE back, and SEA already reports a closed error.

DBSQLOperation now tracks its close and cancel requests until they settle. When a wait or fetch fails with anything other than an OperationStateError, it waits for every request still in flight. If a close or cancel succeeded, it reports OperationStateError CLOSED or CANCELED. Otherwise it rethrows the original error. All of them are awaited because closes and cancels can overlap, for example a cancel followed by session.close(), or a caller's own close() alongside session.close().

Verification, with the kernel at 0b5d0ed:

  • 11 new unit tests in tests/unit/DBSQLOperation.test.ts. 7 fail with the base DBSQLOperation (93a9112): close → closed, with the close settled before or after the poll fails, and with the poll failing on a 409 or on a 404 THTTPException; cancel → canceled; a cancel overlapped by a failed close; a second close overlapping a successful one and failing; and a result fetch failed by close. The two overlap cases also fail when only the latest close or cancel is awaited (f12a671 fails both, 1ffdb2b the duplicate-close one), and the pending-close cases fail when nothing is awaited. The other 4 cover the cases that must keep the original error. npm test (1762 passing), lint, type-check and prettier are clean.
  • Live, using the databricks-driver-test RaceConditionTests (now on its main, from #2082) with the Thrift xfail turned into the hard assertion: CONCURRENT-004 and CONCURRENT-005 pass 3/3 on Thrift and 3/3 on SEA, with the driver at main 91a64ff plus this PR (and earlier at 8ccddd0 plus this PR). Every Thrift CONCURRENT-004 run hit the 409 race and reported The operation was closed by a client. The unmodified tripwire XPASSes against this build, so databricks-driver-test needs a PR that flips it to a hard assertion once this merges.
Spec Thrift SEA
CONCURRENT-004 bug → passes passes (unchanged)
CONCURRENT-005 passes (unchanged) passes (unchanged)

This pull request and its description were written by Isaac.

Closing or cancelling an operation while fetchAll(), fetchChunk() or
finished() waits on it could reject the wait with the server's answer
for the released statement. On Thrift, closing the session while a query
runs makes the in-flight GetOperationStatus return HTTP 409 ("A concurrent
GetOperationStatus request updated the command. Please retry."), which
surfaced as a THTTPException. DBSQLOperation now records a close or cancel
from the moment it is requested and reports such failures as
OperationStateError CLOSED or CANCELED.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall. The new close/cancel flags fix the 409 case as described, and the tests cover the main paths. One low concern: if the in-flight request fails before a close or cancel that then fails, the waiter is told CLOSED/CANCELED while the operation stays open, and the original error is dropped without a log. That case isn't tested.

Comment thread lib/DBSQLOperation.ts Resolved
A status poll or result fetch could fail while the close or cancel that
caused it was still pending. The waiter was then told CLOSED or CANCELED
even if that request went on to fail, leaving the operation open and
dropping the original error. Keep the latest close/cancel request and
await it before choosing the error: report the client's action only if
it succeeded, otherwise rethrow the original error.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

A cancel and a close (e.g. from session.close()) can overlap. With only
the latest request kept, a close that failed on the unfinished statement
made the waiter rethrow the poll error even when the concurrent cancel
succeeded. Keep the close and cancel requests separately and await both.
Add tests for a failed cancel and for the overlap, and the CHANGELOG
reference.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

The existing close test resolved CloseOperation within a few microtasks,
so it passed even without awaiting the close request. Add the
CONCURRENT-004 ordering where the poll fails first and the close then
succeeds, and rename the existing test to the ordering it covers.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

Besides the 409, the server can answer the in-flight GetOperationStatus
with 404 RESOURCE_DOES_NOT_EXIST once the close released the statement.
Pin that shape with a real THTTPException.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low concern. The fix is narrow, the microtask ordering that clientStopError depends on is sound (closeInternal/cancelInternal set their flags before the derived .catch() continuation runs), and the tests cover both the remap path and the cases that keep the original error. Follow-up, not caused by this PR: getSchema()/getMetadata() result-metadata RPCs that fail during a close still pass the raw transport error through.

Comment thread lib/DBSQLOperation.ts Resolved
@vuanhphung vuanhphung closed this Nov 15, 2026
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Reopening: this was auto-closed by GitHub because the merged driver-test PR databricks/databricks-driver-test#2082 referenced it after a closing keyword ("fix: …#609"). It is still awaiting review.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

Two close() calls can overlap too, e.g. a caller's own close() alongside
session.close(). Keeping only the latest request let a failed second
close make the waiter rethrow the poll error while the first close was
still about to succeed. Track every close and cancel request until it
settles and await them all.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

On the review-body follow-up (getSchema()/getMetadata() result-metadata RPCs that fail during a close still pass the raw transport error through): agreed, and it is out of scope here on purpose. This PR covers the wait and the result fetch, which is where fetchAll(), fetchChunk() and finished() spend their time; the CHANGELOG entry names exactly those. The metadata request after the wait is a single short RPC, and fetchAll() never issues it after the first chunk (Thrift hasMore() does no I/O once the result handler exists). Extending the same conversion to getSchema(), getMetadata() and an iterator's first hasMoreRows() is a reasonable follow-up.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Closing: not needed. On main, a caller that retries after the 409 "…Please retry." gets OperationStateError(CLOSED) immediately, without a server call: the operation's closed flag is set before the session closes and failIfClosed() short-circuits. Verified live on Thrift in 30 of 30 runs, and SEA surfaces CLOSED directly. A retry fired synchronously before the close completes can see one 404 RESOURCE_DOES_NOT_EXIST first. The driver-test CONCURRENT-004 spec is being relaxed to accept this retryable-conflict shape.

@vuanhphung vuanhphung closed this Nov 16, 2026

This branch had an error being deployed

1 failed deployment
azure-prod — f69e7888 Deployed Nov 16, 2026 by vuanhphung via e2e-test (24.22.1, 24_22_1) #2058
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant